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This Stufflebeam-Guba CIPP type model for the evaluation of innovations in 
education attempts to maximize the effectiveness of critical decisions through the 
timely reporting of relevant information in a useful form to appropirate levels of 
decision making. Evaluation is thus seen as the combination or effective decisions 
based on timely, relevant information. The system focuses on four classes of 
decisions and is designed to yield four kinds or information to serve those decision 
situations. These four kinds of evaluation are context evaluation, design evaluation, 
process evaluation, and product evaluation. Context evaluation consists of planning 
decisions and the context information that serves them. It deals with the setting of 
priorities and the selection of strategies. Design evaluation entails structuring 
decisions which depend on design information. In this phase, objectives are specified 
and means to attain them are selected. Process evaluation deals with the possible 
need to restructure the program after results of pilot testing and previous 
evaluations are in. Product evaluation considers evidence about the program’s 
effectiveness in attaining its overall goals. Problems in applying this system might 
include identifying decisions and decision makers, timing decisions, identifying relevant 
information, and reporting information in a useful form. (TT) 
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Introduction 



This paper is concerned with systematic evaluation of innovations 
in education. Other kinds of evaluation are important, but they are 
out of the scope of this presentation. The evaluation system which this 
paper is about refers to innovative projects that are planned, designed, 
and validated. The paper rests on several assumptions that will not be 
argued, except indirectly in an explanation of the evaluation system. 

The most important assumptions are listed below. 

*1. Evaluation is different from measurement. 

2. Research designs are inadequate and inappropriate for most 
evaluation purposes. 

3. Systematic evaluation is needed before objectives of an innovative 
project are designed. 

4. Effective evaluation cannot be performed solely by either an 
outside organization or a separate division within an organization. 

It should not be concluded from the preceding assumptions that reliability 
and validity in evaluation can be compromised, but rather that new methods are 
needed to effect them. 

The presentation will first describe an evaluation system and how it 
functions. Then attention is turned to problems of application of the system. 



An Evaluation System 

The evaluation system described here is an adaptation, developed by 
the Southwest Educational Development Laboratory, of the Stuff lebeam- 
Guba CIPP model.* ** The major objective of this system is to maximize the 
effectiveness of critical decisions that are made in an organization 
through the timely reporting of relevant information in a useful form to 
appropriate levels of decision-making, in order to optimize planning and 
development activities. Thus, evaluation is seen to be the combination 
of effective decisions based on timely, relevant information. The system 
focuses on four general classes of decisions and is designed to yield 
four general kinds of information to serve those decision situations. The 
four kinds of evaluation, called context evaluation, design evaluation, 
process evaluation, and product evaluation, are described as follows. 



*Those interested in arguments for these assumptions are referred to the 
symposium on "The World of Evaluation Needs Reshaping," AERA, Saturday morning 
February 8. 

** Developed by Dan Stuff lebeam of Ohio State University and Egon Cuba of 
Indiana University. 



Context Evaluation 



Context evaluation consists of planning decisions and context infor- 
mation that serves them. In planning a new project two important kinds of 
planning decisions are encountered. 

1. The first is the selection of problem components that will be 
attacked, e.g. , setting priorites. 

2. A second kind of planning decision is the selection of a strategy 
(or strategies) that would be used to attack the given problem(s) . 

To serve problem priority decisions (the first kind) , the information 
needs include knowledge both of constraints of the organization and of the 
conditions that exist in the general problem area, assuming that not all 
components of the problem could be attacked initially. The information 
gathering tasks are to identify the set of problems and problem compoi.^nts 
which might be attacked in order that priorities can be established. An 
important weakness in many planning phases is the failure to identify the 
relatively few problems or problem components that effectively can be 
attacked, thus identifying others that are not primarily under attack or 
which will be left to a later date. 

In the case of strategy selection (the second kind of decisions) , 
the information needs are to identify resource constraints and to describe 
general strategies, methods, or approaches that might be used to attack 
the problems in question. Again the decision task is to select from among 

several available strategies one or more that are most appropriate in 
view of the resource constraints, thus consciously rejecting other strategies 

that might have been used. 

The major sources of context information are research surveys, the 
literature, and expert opinion. For example, suppose a project is being 
planned in the general problem area of doing something about the needs of 
young children. In identifying organizational constraints, it might be 
determined that the organization is best suited to educational attacks. 

Then by conducting research surveys on educational problems of young children, 
by reviewing research literature and activities of early childhood education 
projects, and/or by obtaining advice of experts in child development, an 
array of general problem areas might be outlined which could be attacked. 

The priority decision might limit the problems to those of intellectual 
needs and language needs. Then, after identifying the resource constraints 
(that is, the amount of money, the availability of personnel, and the length 
of time), several strategies might be considered that would better meet 
the intellectual and language needs of young children. These might vary from 
instruction for the mother of the child to setting up a boarding school 
for children. However, by reviewing both strategies that have been used 
by others and those that can be proposed by experts, the decision-makers 
can select a strategy that would be feasible. It would need to be considered 
in the light of resource constraints and support a reasonably good rationale 
for its selection over other strategies. 

Design Evaluation 

Design evaluation entails structuring decisions which depend on design 
information. Structuring decisions are made in the same manner as a blue- 
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print or set of specifications is designed. Both ends and means need to be 
structured in detail; that is, the objectives need to be specified opera- 
tionally if possible, and activities or means of attaining them need to be 
specified. Information needs include evidence from content or behavioral 
fields, combined with the knowledge and ingenuity of experts. 

To continue the example, suppose a strategy in early childhood educa- 
tion is that of instructing mothers, who would in turn improve the intellec- 
tual and language development of their children. The general strategy would 
postulate long-range goals for intellectual and language development of the 
children. Therefore, the task of structuring would focus on objectives and 
activities for training the mother to interact with her child. Research 
evidence about effective instructions of mothers, information from other 
projects, and the knowledge and ingenuity of experts in learning theory, 
child development, etc. would be important information needs in making the 
structuring decisions. 

Process Evaluation 



After a design has been structured and is put on trial, often called the 
pilot test, restructuring decisions are faced. Restructuring decisions are 
based on process information. Process information consists of the evidence 
needed to determine effectiveness in attaining the objectives and other 
information about how the test is being conducted. Although evidence that 
determines the extent to which objectives are attained is important, it 
often is not helpful in suggesting the kinds of revisions that might be 
made when objectives are not attained satisfactorily during the trial. 

Other information, such as the length of time an activity requires and 
impressionistic information about the trial, can be useful in determining 
in what ways the design may be restructured. The major question is what 
information would be useful in determining how to restructure in order to 
get a design that will meet its specifications during the trial. Designers 
and evaluators need to consider very carefully what information would cause 
them to change the original design, then plan a system of information 
gathering that will obtain the needed information. This might entail 
gathering baseline observations or test results against which effects at 
given times can be compared, as x^ell as comparing the stated objectives 
with observed effects. It may or may not entail observations or tests with 
comparison or control groups , depending on the length of time for each 
trial. Such controls would only be necessary if there was reason to suspect 
that other subjects might attain the objectives through maturation or some 
other factor involved in the passing of time. This may not be the case 
during pilot test when small increments of time are involved. 

The point can be illustrated by extension of the example previously 
used. An objective of instruction of mothers might be that each mother 
would spend at least one hour each day interacting with her children. The 
designed activity might be a discussion emphasizing the importance of such 
interaction combined with a role-playing demonstration. Evidence from 
follow-up observations might reveal that the objectives were not adequately 
obtained. Therefore, impressionistic- information from the discussion leader 
and the role-playing leader, as well as from the mothers involved, might 
reveal the reason for failure and suggest ways to redesign the session. 

The use of process information obtained during design, trial, and redesign 
enables components of a larger program to be built. 



Product Evaluation 



After components of a design have been tested* they can be put together 
in a program for a product or field test. Since this is the first full- 
cycle test, the major decisions faced are whether to recycle through another 
full-scale field test. The information needed, called product information, 
entails not only evidence about effectiveness in attaining short- and long- 
range goals, but also effectiveness over a several-month or year time 
period compared with that of another program or strategy. The attempt here 
is to obtain convincing evidence that the program will attain its objectives 
and, moreover, that those objectives are worthy and reasonable when compared 
with other methods. Product information should also include the resources 
and cost needed to make the program effective. 

In the example, mothers would attain (or fail to attain) objectives 
set forth for them in the time prescribed; and effects on their children 
would become evident through evidence that was accumulated. This evidence 
on the children would be used to validate the original strategy when comparing 
effectiveness with other programs or the situation as it existed without such 
a program. 

_ Analysis 

It may be argued that this system is too extensive for certain evaluation 
purposes. Obviously it is foolish to spend $10,000 evaluating a $5,000 project; 
but the contention here is that the use of this system will enable the designers 
of a project, whether it be a $10,000 project for six months or a $1,000,000 
project for one year, to plan, design, and test more systematically and 
more efficiently than they could with a less systematic approach. The 
time and resource constraints (including financial) , identified in the 
first stages of planning, should be the guide to the degree of sophistica- 
tion one might reach in applying the model through each phase. However, 
a systematic approach in identifying major decisions and information on 
which those decisions can be based will produce a more effective design 
with the least amount of effort. Figure 1 summarizes in a matrix the nature 
of decisions, the kinds of information, and the sources of information called 
for in the system. 

A flow chart is presented in Figure 2, which details the use of the 
evaluation system. The initial context evaluation phase is described in 
any good proposal. Information is drawn from the "educational domain, 1 that 
is, the general setting and the world of research and experimentation. But 
many proposals are designed to "get the money," and thus planning may be 
"skimpy." This is understandable and satisfactory in the light of communi- 
cation about funding dates and costs for proposals, which often impose un- 
realistic time constraints. However, in such cases, a context evaluation 
should be conducted following submission of the proposal, since poor planning 
(inadequate definition of problem areas and selection of strategy) spells 
the early doom of many projects. 

Although not specifically shown in the flow chart, the information 
accumulated and used at each stage is passed on to the next stage and 
used along with new information that is obtained. That is, the information 
is always cumulative. It can be noted that as each kind of information is 
collected (e.g., design, process, product), provision is made to return to 
the "educational domain" to supplement the information. That is, new research 
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and emerging innovations are : oaitored and feu » to the system. All the 
information, after its initial use, is mainl.'ired in the continuing context 
information component. 



Problems of Application 

Having described the evaluation system and outlined the way it 
functions in an organization, let us turn our attention to the problems 
that are encountered in its application. The key to the effective oper- 
ation of the system is that decisions are identified along with persons 
who ..re involved in the decision process. Then a systematic attempt is 
made to get relevant, timely information in a useable form to the decision- 
makers. The assumption is that the most effective decisions are those based 
on the best information. Hence, the task is to get the best information to 
the decision-makers in the time that is available. However, in operation 
this task poses some serious problems. 

Identifying Decisions 

Decisions that are faced are not always easily recognized. Often 
decision-makers themselves are not fully aware of the decisions they may 
face. In introducing new information, the evaluation system may focus 
attention on decisions that were not previously considered. Hence, the 
system must provide persons who are in contact with key decision-makers 
and are continually alert to decisions that will be faced. 

Another problem in identifying decisions and their nature is that 
decision criteria may change as time passes. New developments occur; new 
information is obtained; conditions change as time goes by. Any one of 
these can cause new criteria to appear or old ones to be of no effect. 

Hence, the system must provide for a continual reassessment of criteria that 
may affect decisions. 

The passing of time may also cause constraints to change . Since there 
is always some lag between the time when decisions are identified and the 
time when information is collected, processed, and reported, the system 
must continually be alert for changes in constraints that might change the 
basis on which decisions will be made. 

Identifying Decision-Makers 

Another problem is the identification of persons involved in the 
decision process. These include not only those who have final authority 
in making decisions but others involved in the decision process who may 
influence the final decision-maker. Typically, the decision process in 
an organization involves a complex network of persons who have varying 
degrees of influence on the one who may have constituted authority to 
make any given decision. Hence, it may be useless to get information to 
the recognized, final decision-maker, in that he either may have little 
time for considering the information or may rely heavily on the judgment 
and recommendation of other people. Therefore, the evaluation system must 
identify the key persons involved in any strategic decision and make arrange- 
ment for getting necessary information to these people. 



Timing of Decision.- . 

The best information is of utterly no use if it does not arrive in time 
to base a decision on it. Therefore, the key for the operation of an eval- 
uation system is to get the best information possible in the time that is 
allowed. Of course, it is possible to postpone the time of the decision, but 
often such a delay is not possible. Hence, the system must respond to the 
time when critical decisions will be made and yield the information needed 
in time for it to be considered. 



Identifying Relevant Information 

It is not enough for evaluators to decide what information would be 
best on which to base the decision. Cues must be taken from the decision- 
makers as to what information is relevant to their decision tasks. It is 
useless to force sophisticated information upon a decision-maker who fails 
to see its relevance, since he will ultimately disregard it in favor of more 
understandable, if less relevant, information. The system can be designed 
to educate decision-makers to the usefulness of certain kinds of information, 
but the final criterion must be that the decision-maker considers the infor- 
mation relevant. Otherwise, the best information will have little, if 
any, effect on the decision. 

Reporting in a Useful Form 

Another problem related to die relevance of information is to get the 
information to the appropriate decision— maker.** i.n a form that is most useful 
to them. This entails not onlv varying the degree of sophistication but 
also the degree: of speet i i<* i ty <>i reports. The criteria must include the 
length of the time die i<'oj s ion-maker will likely have to consider the infor- 
mation as well as his competence in understanding the terminology and techniques 
used to present the information. Thus, the same information may be presented 
in several different forms to different decision-makers. 



It is obvious from the preceding discussion that communication and inter- 
action with key decision-makers is a cornerstone on which effective evaluation 
rests. We tend to make, ninny unwarranted assumptions about the effectiveness 
of our communi cations . One of the hazards of .written communication is that 
the writer has little control over who will read Hs paper, what psychological 
set they will have as rhe\ read it, or how they *. .11 interpret it. Furthermore, 
he has no chance to internet or clarify his meaning or intent with many of the 
readers. Therefore, the more visual and oral cues and face-to-face interaction 
that can accompany hi'*, cutton communications, the more chance he has of being 
understood. Such research as we. have on communication suggests that we are 
more likely to fail to he understood than to communicate effectively if 
we depend on any single sensory perception. 



The* real test o i this system Is the extent 
makers and evaJ.ua tors to the nature ol decisions 
decisions with relevant, t finely information in a 



to which it alerts decision- 
faced and services those 
useable form. 




